DRILL-6049: Misc. hygiene and code cleanup changes #1085

paul-rogers · 2018-01-08T22:56:04Z

Collection of "hygiene" changes arising from the vector column writer project. This represents the set of changes outside of the core mechanism so that reviews can be done in multiple PRs rather than a single monster PR that mixes generic and core work.

paul-rogers · 2018-01-08T23:26:42Z

Error Handling

The column writer work introduces a new scan operator mechanism. Part of that mechanism is to standardize error handling. Part of that work is, in turn, to add a a couple new error types to UserError to better categorize errors.

Star Column

Organized our many references to the "star column" to make it more readily available to the new scan framework.

Date/Time Utilities

The various date/time vectors have very handy methods to convert the vector representation into Joda objects. This PR pulls that code out into a utility class that can be used in other places. This change uncovered a flaw in JDBC: that the existing date/time utilities were not included in the JDBC jar, causing failures. This PR fixes that.

Deprecate OptionSet

In a previous set of PRs we've been assembling a sub-operator test framework. This PR continues that work by deprecating the prior OptionSet class. (OptionSet was meant to work around limitations in the original OptionManager, but those issues have since been resolved by all the great work that Jyothsna and Tim have done. Since OptionSet now no longer serves a useful purpose, this PR removes it.)

Deprecate StatsWriter

Similarly, the StatsWriter class added a stats mechanism that can be used in sub-operator tests. Further experience has shown we can make the full OperatorStats work in this context, so StatsWriter is deprecated in this PR.

Fix for Close of Sort Operator

A bit of code was reshuffled to close the operator context in the correct order in the sort operator in both production and unit tests.

Swap Containers

Added code to swap buffers between two containers. Builds on code added earlier to exchange buffers between two vectors. (Used in scan operator, to be added later.)

CSV Column Header Error Handling

Changed CSV to use UserException to report errors instead of an ad-hoc HeaderError exception.

Test Tools

Add a number of services to the ClusterFixture family of test classes. The key new feature is the ability to read the results of a multi-batch query as a sequence of RowSet objects so we can use the row set tools to verify results.

Union Vector

The union vector is a rather odd duck: it works, but is not widely supported in Drill. It stores its "type" members in two ways: in an internal map and in member variables. The member variable format makes it hard to treat the types generically. This PR keeps the serialized format (storing vectors in an internal map) but changes the member variable format to use an array of type vectors rather than individual variables. The result is that generic access (for the new Union column writer) is much easier.

Materialized Field in Vectors

Vectors provide a MaterializedField to hold their metadata. The original idea seems to be that a MaterializedField is immutable. This worked great for simple scalar types. But, code evolved to change the MaterializedField for maps (to add map members), for Unions (to add union members) and so on.

For container types (Map, Repeated Map, Union, List, Repeated List), the "parent" MaterializedField holds onto a copy of the child field. But, some parents, when they receive new members, throw away their old MaterializedField and create a new one. The result is that, if we have the structure map(map(a, b)), when we add b to the inner map, the parent ends up pointing to an older version of the child schema, not the updated one.

This PR fixes at least some of those issues in the Union vector.

JUnit Upgrade

Upgrades JUnit from 4.11 to 4.12 to make use of a handy annotation that disables timeouts when debugging.

Generic Cleanup

Added comments, fixed indentation and other minor changes.

arina-ielchiieva · 2018-01-12T12:14:20Z

common/src/main/java/org/apache/drill/common/exceptions/UserException.java

   */
-
+  @Deprecated


Deprecated means that we might remove it in future. But since you mention that it is still ok to use in certain cases, I am not sure that such annotation is appropriate.

Good catch. Merge error. It is deprecated in master; I have undeprecated it in my branch...

arina-ielchiieva · 2018-01-12T12:20:07Z

common/src/main/java/org/apache/drill/common/types/Types.java

+    return copy1.equals(copy2);
+  }
+
+  public static String typeKey(MinorType type) {


Why we need this method? It looks like it is never used? If it's intended for future use please add java doc then.

arina-ielchiieva · 2018-01-12T12:22:37Z

common/src/test/java/org/apache/drill/test/DrillTest.java

@@ -54,8 +56,7 @@
  static MemWatcher memWatcher;
  static String className;

-  @Rule public final TestRule TIMEOUT = TestTools.getTimeoutRule(100000);
-
+  @Rule public final TestRule TIMEOUT = new DisableOnDebug(TestTools.getTimeoutRule(100_000));


Great enhancement!

arina-ielchiieva · 2018-01-12T12:38:21Z

exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorStats.java

-  @Override
+  @VisibleForTesting
+  public long getLongStat(MetricDef metric) {
+    return longMetrics.get(metric.metricId());


If metric is absent, will it return 0 or fail?

Presumably fail. But, since this used in testing to ensure that a long metric was, in fact, written, failure is actually a useful result.

arina-ielchiieva · 2018-01-12T12:57:41Z

exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java

@@ -38,20 +38,26 @@
 import org.apache.drill.exec.server.options.SystemOptionManager;
 import org.apache.drill.exec.store.sys.store.provider.LocalPersistentStoreProvider;
 import org.apache.drill.exec.util.GuavaPatcher;
+import org.apache.drill.test.BaseDirTestWatcher;


Starting from this class, github does not highlight Java code. I remember @vdiravka had similar problem, it turned out that he copied code from text editor with some hidden symbols. Please check and fix.

Checked the source. No hidden characters. For me, formatting does appear below this line.

I suspect your browser hit some limit. Large PRs make Safari struggle to display the code. Suggestion: refresh the page. search for this file, and keep going. Perhaps, if you don't expand prior files, your browser will handle the formatting. (GitHub should provide some solution for large PRs...)

paul-rogers

@arina-ielchiieva, thank you for your kind review. Comments were addressed. Looks like GitHub UI prevented you from reviewing some of the files. See the suggestions inline and please try again.

paul-rogers · 2018-01-19T00:48:55Z

common/src/main/java/org/apache/drill/common/exceptions/UserException.java

   */
-
+  @Deprecated


Good catch. Merge error. It is deprecated in master; I have undeprecated it in my branch...

paul-rogers · 2018-01-19T00:51:14Z

common/src/main/java/org/apache/drill/common/types/Types.java

+    return copy1.equals(copy2);
+  }
+
+  public static String typeKey(MinorType type) {


paul-rogers · 2018-01-19T00:53:14Z

exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorStats.java

-  @Override
+  @VisibleForTesting
+  public long getLongStat(MetricDef metric) {
+    return longMetrics.get(metric.metricId());


Presumably fail. But, since this used in testing to ensure that a long metric was, in fact, written, failure is actually a useful result.

paul-rogers · 2018-01-19T00:57:00Z

exec/java-exec/src/test/java/org/apache/drill/exec/ExecTest.java

@@ -38,20 +38,26 @@
 import org.apache.drill.exec.server.options.SystemOptionManager;
 import org.apache.drill.exec.store.sys.store.provider.LocalPersistentStoreProvider;
 import org.apache.drill.exec.util.GuavaPatcher;
+import org.apache.drill.test.BaseDirTestWatcher;


Checked the source. No hidden characters. For me, formatting does appear below this line.

I suspect your browser hit some limit. Large PRs make Safari struggle to display the code. Suggestion: refresh the page. search for this file, and keep going. Perhaps, if you don't expand prior files, your browser will handle the formatting. (GitHub should provide some solution for large PRs...)

arina-ielchiieva · 2018-01-19T09:48:55Z

@paul-rogers, hope this is browser issue but just in case please take a look at UnionVector class to confirm that everything is fine. Also please resolve merge conflicts. Other than that, +1, LGTM.

amansinha100 · 2018-01-22T16:26:40Z

@paul-rogers since the DRILL-3993 (Calcite related changes) went into master last week, this PR would need to be rebased. I can merge it in soon after that.

paul-rogers · 2018-01-24T05:15:23Z

@amansinha100, rebased on master. Ran unit tests. Please run pre-commit tests and let me know if anything is amiss. Once I hear the tests pass, I'll mark this as ready-to-commit. Thanks.

amansinha100 · 2018-01-24T16:11:34Z

Sounds good. I will run the pre-commit tests. Since this PR was previously in a ready-to-commit stage and only needed a rebase, it is fine to mark it ready-to-commit.

close apache#1085

arina-ielchiieva requested changes Jan 12, 2018

View reviewed changes

paul-rogers commented Jan 19, 2018

View reviewed changes

paul-rogers mentioned this pull request Jan 22, 2018

DRILL-6071: Limit batch size for flatten operator #1091

Closed

DRILL-6049: Misc. hygiene and code cleanup changes

51973b4

paul-rogers force-pushed the DRILL-6049 branch from fb0f88a to 51973b4 Compare January 24, 2018 05:13

asfgit closed this in e791ed6 Jan 24, 2018

reudismam pushed a commit to reudismam/drill that referenced this pull request Jan 31, 2018

DRILL-6049: Misc. hygiene and code cleanup changes

ccabba3

close apache#1085

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DRILL-6049: Misc. hygiene and code cleanup changes #1085

DRILL-6049: Misc. hygiene and code cleanup changes #1085

paul-rogers commented Jan 8, 2018

paul-rogers commented Jan 8, 2018

arina-ielchiieva Jan 12, 2018

paul-rogers Jan 19, 2018

arina-ielchiieva Jan 12, 2018

paul-rogers Jan 19, 2018

arina-ielchiieva Jan 12, 2018

arina-ielchiieva Jan 12, 2018

paul-rogers Jan 19, 2018

arina-ielchiieva Jan 12, 2018

paul-rogers Jan 19, 2018

paul-rogers left a comment

paul-rogers Jan 19, 2018

paul-rogers Jan 19, 2018

paul-rogers Jan 19, 2018

paul-rogers Jan 19, 2018

arina-ielchiieva commented Jan 19, 2018

amansinha100 commented Jan 22, 2018

paul-rogers commented Jan 24, 2018

amansinha100 commented Jan 24, 2018

DRILL-6049: Misc. hygiene and code cleanup changes #1085

DRILL-6049: Misc. hygiene and code cleanup changes #1085

Conversation

paul-rogers commented Jan 8, 2018

paul-rogers commented Jan 8, 2018

Error Handling

Star Column

Date/Time Utilities

Deprecate OptionSet

Deprecate StatsWriter

Fix for Close of Sort Operator

Swap Containers

CSV Column Header Error Handling

Test Tools

Union Vector

Materialized Field in Vectors

JUnit Upgrade

Generic Cleanup

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paul-rogers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arina-ielchiieva commented Jan 19, 2018

amansinha100 commented Jan 22, 2018

paul-rogers commented Jan 24, 2018

amansinha100 commented Jan 24, 2018